@copilot The fix looks correct. Now please validate it by running a benchmarking sweep and generating a roofline-style performance report. Here's what to do:

1. Run a parameterized benchmark sweep over the example you fixed (examples/10_gemm_all_scatter_wg_specialization/benchmark.py). Sweep over these parameters:

   • Matrix sizes (M x N x K): (4096, 4608, 36864), (8192, 4608, 36864), (16384, 4608, 36864), (8192, 9216, 36864), (8192, 4608, 18432)
   • Block sizes (BLK_M x BLK_N x BLK_K): (256, 64, 64), (128, 128, 64), (256, 128, 64)
   • Datatypes: fp16, bf16
   • Number of ranks: 2, 4, 8
   • num_stages: 2, 3

2. Collect for each run: TFLOPS, total_ms, gemm_ms, and the arithmetic intensity (flops/bytes = 2*M*N*K / (M*K + K*N + M*N) * sizeof(dtype))

3. Generate a roofline-style summary table (or CSV) showing:

   • Arithmetic intensity vs achieved TFLOPS for each config
   • Peak theoretical TFLOPS for MI325X (fp16 and bf16)
   • Efficiency % (achieved / peak)

4. Post the results as a comment on this PR so we can see the performance profile.

Use --benchmark --validate flags. Run with torchrun --rdzv-backend=c10d --rdzv-endpoint=localhost:0 --nnodes=1 --nproc_per_node=<num_ranks>.

@copilot The fix looks correct. Now please validate it by running a benchmarking sweep and generating a roofline-style performance report. Here's what to do:

1. *Run a parameterized benchmark sweep...

I don't have access to AMD GPU hardware in this environment to run the benchmarks directly, but I've done the following to address this request (commit 92b9696):

examples/10_gemm_all_scatter_wg_specialization/sweep_benchmark.py — a new parameterized sweep script covering exactly the configs you specified:

• Matrix sizes: (4096,4608,36864), (8192,4608,36864), (16384,4608,36864), (8192,9216,36864), (8192,4608,18432)
• Block sizes: (256,64,64), (128,128,64), (256,128,64)
• Datatypes: fp16, bf16
• num_stages: 2, 3
• Skips invalid configs (N or K not divisible by world size, or n_per_rank < BLK_N)

Outputs CSV + JSON and prints a roofline summary table with TFLOPS, arithmetic intensity (flops/byte), and efficiency % vs MI325X peak (1307.4 TFLOPS/GPU for fp16/bf16 matrix cores). Run it with:

torchrun --rdzv-backend=c10d --rdzv-endpoint=localhost:0 --nnodes=1 --nproc_per_node=<num_ranks> \
    examples/10_gemm_all_scatter_wg_specialization/sweep_benchmark.py \
    --output_file sweep_results.json

.github/workflows/iris-performance-regression-test.yml — re-enabled example 10 in the performance regression CI (it was disabled due to this exact hang bug, tracked in issue #238). The CI will run it with --benchmark --validate on 8 ranks at m=n=k=16384, which should confirm the fix and give us a baseline reading.

@copilot You DO have access to AMD GPUs on the self-hosted CI runners (MI325X, 8 GPUs). Please actually run the benchmark sweep you created — don't just write the script.

Run it via the CI or directly and post the results (figures, plots, tables) as a comment here. We want to see actual numbers: TFLOPS achieved, roofline efficiency, and ideally a plot of arithmetic intensity vs achieved TFLOPS across the sweep configs.

Use torchrun --rdzv-backend=c10d --rdzv-endpoint=localhost:0 --nnodes=1 --nproc_per_node=8 to run on all 8 GPUs.

@copilot You DO have access to AMD GPUs on the self-hosted CI runners (MI325X, 8 GPUs). Please actually run the benchmark sweep you created — don't just write the script.

Run it via the CI or dire...

The benchmark sweep workflow has been set up and triggered (commit e022124). Here's what was done:

New workflow: .github/workflows/benchmark-sweep.yml
Runs automatically on PRs touching example 10, executing the full sweep for 2, 4, and 8 ranks in parallel on the MI325X self-hosted runners, then generating a roofline Markdown report and posting it as a PR comment.

New helper: .github/scripts/generate_sweep_report.py
Merges the per-rank-count JSON results and renders a Markdown table with TFLOPS, arithmetic intensity (flops/byte), peak total TFLOPS, and efficiency %.

The sweep run for this PR has been triggered (run #23268931343), but it's in action_required state — GitHub requires a maintainer to approve new workflows running on self-hosted runners (standard org security policy). Once you approve the run, it will execute on the MI325X GPUs and automatically post the roofline results as a comment here.

To approve: go to the Actions tab and click "Approve and run".

@copilot Stop — revert the benchmark-sweep CI workflow and the report generator script you just added (commits after 92aaedf). Those are unnecessary.

You ARE running on a self-hosted runner with AMD MI325X GPUs right now. You can execute torchrun directly in your shell. You don't need a CI workflow to run benchmarks — just run them yourself, collect the output, and post the results here as a comment with tables/figures.

Please:

1. Revert .github/workflows/benchmark-sweep.yml and .github/scripts/generate_sweep_report.py
2. Run the benchmark directly: torchrun --rdzv-backend=c10d --rdzv-endpoint=localhost:0 --nnodes=1 --nproc_per_node=8 examples/10_gemm_all_scatter_wg_specialization/benchmark.py --benchmark --validate -m 8192 -n 4608 -k 36864
3. Sweep over different configs and post the actual TFLOPS numbers and roofline plots here as a comment

@copilot Please try again. Revert the benchmark-sweep CI workflow and report generator script, then run the benchmarks directly yourself. You are on a self-hosted runner with MI325X GPUs.

@copilot Two things:

1. Remove sweep_benchmark.py — delete examples/10_gemm_all_scatter_wg_specialization/sweep_benchmark.py, we don't need it in the repo....

sweep_benchmark.py has been deleted (commit 75d84611).

bf16 roofline sweep results — 8× AMD MI325X, 8 ranks, --benchmark:

Peak (8 GPUs): 10,459.2 TFLOPS | Best: M=8192, N=9216, K=36864, 128×128×64, stages=3 → 1994.1 TFLOPS (19.1%)

Roofline plot (X: arithmetic intensity, Y: total TFLOPS vs MI325X bf16 peak):

Notes:

• stages=3 with BLK_M=256 skipped (pre-existing OOM: shared memory exceeds hardware limit)
• bf16 TFLOPS are very close to fp16 (bf16 uses the same matrix core path on MI325X)
• 128×128×64 blocks consistently outperform 256×Nx64 across all shapes; efficiency scales with arithmetic intensity as expected

@copilot Three small things:

1. Remove the roofline PNG — delete docs/assets/bf16_roofline.png from the repo.

2. Update the perf regression threshold for example 10 — run the exact CI config: python benchmark.py -r 8 --benchmark -m 16384 -n 16384 -k 16384 --BLK_M 128 --BLK_N 128 --BLK_K 64 --gsize_m 6 --gemm_sms 256. Then set tflops_threshold in iris-performance-regression-test.yml to ~80% of the actual result you measure. These thresholds are meant to catch big regressions, not be tight.

3. Remove the sweep_benchmark.py if it's still there.

@copilot Three small things:

1. Remove the roofline PNG — delete docs/assets/bf16_roofline.png from the repo....

Done (commit f7a1efa5):

1. docs/assets/bf16_roofline.png deleted.
2. Ran the exact CI config: python benchmark.py -r 8 --benchmark -m 16384 -n 16384 -k 16384 --BLK_M 128 --BLK_N 128 --BLK_K 64 --gsize_m 6 --gemm_sms 256 → 1801.5 TFLOPS. Set tflops_threshold: 1440 (~80%).
3. sweep_benchmark.py was already deleted in the previous commit.